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Assessing Learner Strategies Using Computers: 
New Insights and Limitations 

Hsien-Chin Liou 
Foreign Languages, National Tsing Hua University 
Paper Presented at American Association For Applied Linguistics Annual Meeting 

Long Beacli, Marcli 25-28, 1995 

ABSTRACT 

In computer-assisted language learning (CALL) environments, exchanges 
between (a) tlie input directly given in the material, or optional resources upon 
learner request, and (b) what the learner already knows become interesting as 
the electronic materials proliferate. This paper discusses the assessment of 
learner strategies in the CALL context, and the pros and cons. It first reviews 
the empirical studies, including two of the author's recent projects, which have 
used the computer to record the interactive process where L2 learners 
demonstrate different on-line language learning behavior, operationalized as 
types of learner strategies* The available literature has covered the areas of 
grammar learning, writing, reading process, and listening and viewing (video) 
comprehension. Types of strate^s included cognitive and metacognitive 
strategies such as resourcing, monitoring, practicing, or self-evaluation. The 
data help applied linguists to better understand such situated learning, i. e., in 
CALL, by providing insights into the interaction between input and learners. 
The paper then illustrates the detailed process of data gathering, unitization, 
and interpretation. The findings and data collection techniques provide 
researchers in other fields a new horizon where different sources of L2 learner 
data can be triangulated to help uncover the SLA phenomena. Advantages of 
the powerful computer recording capacities lie in its accuracy, real-time 
immediacy, reliability, and compact storage space. However, the mechanical 
method has its limitations. It is argued that computer-based data collection 
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methods are meant to complement other traditional methods, especially when 
the former are constrained* 

L INTRODUCTION 

The state of the art of the learner strategy research tell us that good learners are 
believed to be able to use effective strategies such as making inferences, using 
deduction, or seeking clarification (e. g-, O'Malley & Chamot, 1990; Oxford, 1990; 
Weinstein, et al-, 1988; Wenden & Rubin, 1987). Nevertiieless, the field has been 
facing challenges in research methods* Namely, how to collect valid data that show 
what learners are reaUy doing on language learning tasks. Rees-Miller (1993) raised 
the issue in a recent TESOL Quarterly article: •'it is questionable whetiier they 
[cognitive learning strategies] can be specified in terms of observable, specific, 
universal behaviors tiiat could be taught to or assessed in students •••in order to know 
whether teaching has been successful, teachers must be able to observe students 
performing the technique and how that the behavior observed indicates that students, 
from whatever culture, are practicing the target strategy** (p. 681)^ Indeed, both the 
language teachers and researchers need to observe learners in a way which is consistent, 
observable, specific, and universal so that the evidence can be easily cross-validated and 
thus disseminated^ 

O'Malley and Chamot (1990) have raised a framework for strategy data 
collection of which the elements and considerations are summarized in Table !• 
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Table 1 Framework of Strategy Data Collection 



Six Elements 



Detailed Considerations 



Objective of data collection 



1. declarative/procedural knowledge of 
strategy 

2. overt/covert strategies 

3. types of strategy 



Language task 



1. L1/L2 

2. modalities 

3. specificity 



Temporal relationship 



1. simultaneous introspection 

2. immediate retrospection 
3« delayed retrospection 

4. predicative introspection 



Informant training 



Elicitation procedures 



h language 

2. degree of structure 

3. oral/ written response 



Individual/Group data 
collection 



In conducting strategy research, issues concerning validity and reliability of 
methods, more importantly, have been raised among common ways for assessing 
strategies- The classroom observation skill used in early good learner research can 
record only student's overt behavior. Teachers tend to focus on product than on process 
and need extra efforts to verify the observers' interpretations of l«jmer strategies; thus, 
the approach may be tedious and unproductive. Further, Naiman et al. (1975) pointed 
out that very few learning techniques could be overtly displayed in the classroom. The 
class observation approach is limited to students who speak up, but says nothing about 
those who remain silent. Further, consistency of observation is hard to maintain. Such 
research may require two observers to ensure the reliability. A more difficult problem 
is that we do not know how typical a learner's observed behavior is. In other words, 
are the observational data consistent across day one and day n, or are they just 
idiosyncratic which happen in day n? The self-report methods-including introspective, 
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retrospective reporting, and think-aloud-rely heavily on student's ability to observe and 
report on their learning, although they can be done easily and usually obtain useful 
information about learner strategies* Among them, the common strategy questionnaire 
method further has the drawback of asking respondents to imagine a situation mostty 
using generalized statements and determine what they would do in it. The responses 
may be based on imagination when students probably idealize what they would be- 
labelling themselves* On the other hand, the questionnaire designers may have already 
had a theory in mind beforehand to design the questionnaire, or interview questions- 
Thus, the responses may not be elicited or interpreted objectively (Ericsson & Simon, 
1984)- 

The tliink-aloud method, popular recently, can avoid some of the pitfalls 
mentioned above* In thinking aloud, the subjects actually perform a task and report 
what they are doing while they are doing it, or right after they have finished* It, 
however, has to be carried out with care because it itself has some inherent weaknesses* 
For instance, the think-aloiid protocol would be poor when the reporting method is 
unsatisfactory: information not attended to, only part of the information actually 
reported, or only part of the information retained and thus retrievable* 

Advantages of the powerful computer recording capacities lie in its accuracy, 
real-time immediacy, reliability, and compact storage space* Some scholars (to be 
enumerated shortly) have used the computer to record learning process while learners 
are using the computer software for language learning purposes* There are three 
advantages of computer log data* First, it can record learner behavior over different 
sessions systematically and consistently so it is easy to obtain average performance* 
Second, the data are not based on imagination; they are recorded while learners are 
actually using the learning materials. Last, the data show observable behavior, and 
thus, they can be universal as long as learners across countries or cultures can have 
access to the same CALL materials* Limitations of assessing strategies using computers 
will be discussed in the following section later* 

Cohen (1987) used six factors to contrast common methods mentioned above* 
Comparing the think-aloud method and computer log tracking based on the six factors 
proposed by Cohen (1987), we can see the results in Table 2. 
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Table 2 romparison of Think-aloud and Comput<>r Tracking Methods 



1 factors to be compared 


think-aloud data 


computer log data 


1 # of participant 


individual+investigator 


large group w/o 
investigator 


1 research context 


other context 


class or other with 
computer 


recency of event 


high 


high 


mode of 

elicitation&response 


oral 


oral or written 


formality 


low 


flexible (determined by 
program design) 


degree of intervention 


low 


almost none (controlled 
by program design) 



For the number of participants involved, the computer tracking seems more efficient 
because it can allow many subjects at a time as long as the machines are available. 
Wherever the CALL materials for language classes are available, the computer tracking 
program can work inside or outside the class, whereas the think-aloud normally is 
conducted outside of the class because it is done one by one. Both of them have high 
recency of the language learning events. While the think-loud is elicited in oral mode, 
the computer tracking facilities allow both oral and written modes, the formality of the 
think-loud is low, but the computer program is flexible depending on how it is 
designed. Normally the think-loud has low degree of intervention, but the computer 
program can have none which is again controlled by the program designers. 

n. ASSESSING STRATEGIES USING COMPUTERS 

In this section, a review of literature which summarizes language learning 
research that has used computers as a tool is discussed. Then, the procedures of using 
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the computer tools to collect and interpret strategy data are illustrated through two of 
the author's recent research studies. Last, the limitations of the method are discussed 
with future perspectives. 

Review of Literature on Strategy Research In CALL 

In computer-assisted language learning (CALL), exchanges betv/een (a) the input 
directly given in the material, or optional resources upon learner request, and (b) what 
the learner already knows become interesting as the electronic materials proliferate. Use 
of the optional resources, or on-line help can be operationalized as evidence of use of 
a kind of cognitive strategies, resourcing (Oxford, 1990; Chapelle & Mizuno, 1989). 
Resourcing is an important strategy investigated because this is an inherent advantage 
in CALL materials. As a desired feature in designing electronic instructional media, 
it immediately assists learners during the process. But whether on-line help is useful, 
how often learners tend to use it, what kind of help is crucial for a particular type of 
task, and how the help influences language learning remain unclear to computer-assisted 
language learning (CALL) researchers and developers. Several studies of on-line 
behavior and strategies have been conducted as summarized in Table 3. Back in 
PLATO IV period, Curtin et al. (1981) initiated such research. They recorded student 
users' log key presses, judged answers, requests for help, and working time per 
session. They suggested that the on-line data helped predict successful learners' 
performance, but for weak learners, further investigation is needed. Garrett (1982) 
asked subjects to work on PLATO courseware semester long and recorded their 
performance, believing that the computer tracking facilities make classroom observation 
convenient and precise through the simultaneous collection of data from several 
individual students. Additionally, she used methods that combined classroom 
observation, teacher hunches, and on-line data to study German interlanguage 
development of English native speakers. Last, she proposed a system that conducts 
ongoing classroom research that feeds directly and immediately back into classroom 
teaching and learning. Later, Jamieson and Chapelle (1987) inferred three types of on- 
line data as language learning strategies in an audio dictation lesson and a spelling 
computer lesson among the PLATO ESL courseware. The amount of time elapsed 
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between the end of the audio input and the time that the student pressed the first key 
to begin to answer was inferred as a strategy of advance preparation. At that time, they 
did not consider the time when subjects were idle without doing any learning. 
Monitoring output was inferred by the editing tasks subjects did in their answers. 
Monitoring input was the number of times subjects chose to have audio clues repeated. 
They found that two cognitive styles, impulsivity and field independence, were both 
independentty related to fast response time. The field independent subject was a fast and 
accurate worker who edited his response; an impulsive subject worked quickly, but a 
reflective subject listened more to audio cues in the courseware and worked slowly. 
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Table 3 Summary of the Studies which Used Computer Tracking Method 



1 studies 


language skill or 
language+ 


________ 

♦FOCUS or strategies 
investigated 


cognitive 


metacognitive 


1 Curtin et 
1 al. 1981 


+Russian 


♦error rate & 
time on task 




1 Garrett, 
1 1982 


+German 


♦interlanguage 
development 




Chapelle& 

Jamieson, 

1987 


spelling, 
dictation 




advanced 
preparation, 
monitor input, 
monitor output 


Chapelle& 

Mizuno, 

1989 


grammar 


practice, 
resource 
(online 
help) 


self- 
management 
self-evaluaUon 
self-monitor 


Jamieson et 
al. 1992 


reading 


various (use 

questionnaire 

data) 


various 


Bland et al. 
1990 


writing 


♦language 
development 




Gildea et 
al. 1990 


vocabulary 


various 

resourcing 

strategies 




Chou, 1992 


reading 


skim 

organize 

macro-textual 




Hsu, 1994 


listening 


various 

resourcing 

strategies 




Liou, 1994a 


reading 


read local 
spots, 
post-read, 
pre-reau 
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viewing 


vdnous 








resourcing 








Strategies 





+ those not mentioned are for ESL/EFL learning and when language is mentioned, it 
targets at various skills* 

* when strategy is not mentioned, the study is focused on other language learning 
issues* 

Chapelle and Mizuno (1989) investigated how ESL students used strategies in 
a leamer-^ntrolled grammar lesson at a VAX system. Different students' on-line 
behaviors were again operationalized as strategies while using their grammar CALL 
lesson, based on language learning theories. For example, the self-management strategy 
was inferred by students* use of feedback messages to pursue plans. Results indicated 
that some students did use various on-line strategies but they did not always use the 
optimal strategies. No significant group differences were found among different learners 
of two proficiency levels. They suggest that the type of data collected in the study is 
essential for understanding second language acquisition and developing intelligent 
courseware which is sensitive to students' needs. 

Jamieson et al. (1992) collected various types of student data in order to classify 
learners into successes, failures, and dropouts in CALL context. The data tSiey used 
included notes subjects took, recall, and recognition test scores in an on-line reading 
lesson, and attitudes. In addition, they collected data of individual student characteristics 
(age, sex, first languages, second languages, field def»endence/independence measures— 
a cognitive learning style), learning strategy use (by a questionnaire), and course 
information (various kinds of grades, and credits). They used the discriminant function 
analysis as the multivariate statistic to investigate all the data, and successfully 
differentiated the three groups of learners. They suggest that such research can help 
at-risk learners, and strategy training can be programmed into the computer courseware. 

Hsu, Chapelle, and Thompson (1993) sought evidence for learner use of 
exploration strategy, experimenting and hypothesis-testing about the target language. 
Results indicated that learners explored the CALL environment in a routine way, but 
failed to creatively explore the morphosyntactic possibilities in their grammar 
courseware. As for learner group differences, routine exploration was correlated 
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positively with attitudes of intermediate learners, but negatively with the advanced 
group. On a French writing task, Bland, et al. (1990) analyzed subjects* request of 
glossary help and documented types of help employed as evidence of the learner 
language development. The types of on-line help request are classified as (a) token 
matching for complex morphological queries, phrase search, and search for lexical 
representation of grammatical concept, (b) type matching for base form and 
grammatical relationship searches, and (c) relexicalization using the first language or 
the second language (L2) paths for syntactic or semantic circumlocution. 

Gildea, et al. (1990) launched a series of experiments to investigate how 
contextual enrichment in interactive videodisc material can help vocabulary learning for 
preschool children. They were not driven by perverse affection for modem technology 
but feeing that simpler instnictional method was not enough. Important findings 
regarding types of enrichment in definitions, pictures, and illustrative sentences are: 

1. Children did know when to initiate look-up, but did not know 

what information to ask for. 

2. The children with help of illustrative sentences outperformed all 

others on multiple-choice tests. Definitions added very littie, 
and actually detract from performance over time. 

3. Pictures significantiy improved sentence production for many 

words. 

4. Helpful contexts for learning, i. e., effects for various kinds of 

help, were not additive, (p. 25) 
In EFL context, Chou (1992) investigated the effects of four kinds of help in 
a hypertext reading program: text only (no help), text with on-line vocabulary help, text 
with on-line sentence structural analysis, and text with strategy help (strategies of 
skimming, organization, guidelines to each paragraph, and supplementary information). 
He then randomly divided 39 high schoolers into four groups to work on the four 
versions. He found that types of help did not make a difference on reading 
comprehension among groups of subjects. His questionnaire data show that subjects 
think the vocabulary help was the most useful, and was used the most frequently. Hsu 
(1994) in her dissertation project found that text reinforcement type modifications in a 
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CD-ROM English listening material were most effective for learner improvement in 
comprehension; the oral repetition help was the second effective. The dictionary help 
was founil ineffective because the subjects were at very beginning level who could not 
make good use of such online help. Li (1994) investigated whether active or passive 
on-line help in an IVD unit was beneficial and found that active online help facilities 
were more frequently used; the other types of help were not used because subjects were 
not aware of their existence. He suggests that learners of French did need active 
guidance in using CALL material. Liou (1994a) and Liou (1994b) will be discussed in 
the following section. 

Illustration of the Procedures: Examples 

Twenty EFL college students participated in the study (for a detail report of the 
study on quantitative analyses, see Liou, 1994b). While subjects were viewing a video 
clip and answering questions posed in the program, their on-line history of help 
requests was recorded in a record using a database, dBASE format as shown in Table 
4. The design was itriplemented in the courseware beforehand because the author had 
research agendas in mind. 

Under BACKWARD, the numbers represent the record that a particular segment 
of the video was recalled and replayed upon request. The data in the record reflect 
quantity and quality of strategies use based on different help requests. Each record in 
our study has 13 fields as in Table 4. 
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Table 4 A Record in the Computer Log File 



ID NUMBER= 821201 
NAME= Sherry 
DATE= Thu Dec/ 02 1993 
END_FRAME= 21688 
PAUSED 6789 12099 

BACKWARD = (6824-6700)(l 1866-1 1798)(19339-18876) video frame range 

CHINESE= (7890-ON)(12980-OFF) 

ENGLISH= (5498-ON)(6588-OFF)(12390-ON) 

GIST= 19058 19558 

BACK_INFO= 19058 

REPEAT^' 19058 19058 19588 19588 

REPEAT_P_SENT= 19058 19588 

WORDS = (19058--ectoplasmic) 

We can re-construct the input processing process based on the record. Since the 
subject finished at the point of frame number 21688, we could check our data to know 
that she quit from the unit after she fmished question item 28. She paused twice at 
frame numbers 6789 and 12099, replayed the video from frame numbers 6824-6700 and 
the other two ranges. At frame number 7890, she asked for the Chinese script help 
and turned the help off at 12980. She asked the English help at different points and 
left it on from 12390 till the end, 21688. She requested GIST help twice at 19058 and 
19588 (a text summary for each video clip). Apparently she was stuck there by the 
word, "ectoplasmic" as indicated by the word checked, repetition of the sentence twice 
(REPEAT), and repetition of the previous utterance once (REPEAT_P_SENT). She 
used the two kinds of repetition help. Repeat and P_sent, at 49588 as well. She 
requested BACKJNFO help at 19058 (to get extra information for the setting otiier 
than explanation or content of the utterances in the video). Word and Idiom searches 
were saved under tiie field of WORDS. 

It was found that the occasion when subjects looked up the unknown words 
happened only after they requested the English script help. This makes sense for if one 
does not know the meaning of an acoustic form, he ur she rarely can spell the woid 
out. 
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In the later data analysis, all the frame ranges in the data were standardized by 
numbers of comprehension question items covered within* Because each individual 
frame number is too small as a functional unit, whereas the laserdisk was not coded 
by the time such as minute or second unit* A logical unit is the frame range from one 
question to another one* That is, 54 units. Note the unit is categorical, not scalar 
because the frame ranges among question items vary in video length. This is one way 
to unitize the data because the video information subjects have accessed has seldom 
been studied and its units are hard to quantify. To fulfill language learning purpose, 
we devised such a way to unitize the data. 

Concerning the levels of functions and linguistic or processing factors in the 
computer learning material, further analyses can be made based on the two variables, 
together with the actual help requests made by language learners. 

In this specific computer program, on4ine help has two function levels: the 
video control level, and the comprehension level. In the first level, video '^-ontrol, two 
functions may facilitate input processing: PAUSE and BACKWARD. The PAUSE 
retains visual information of the video, and provides a way to gain more processing 
time. On the other hand, the BACKWARD enables re-access of both visual and audio 
input in larger chunks (than repeating only one utterance), so it is used the most often 
among the nine functions. On the second level, similar comparisons can be made. It 
was found that both the Chinese and English script help functions have high frequency 
of use because they provide all audio, visual, and textual information. The English 
script gets higher frequency because our subjects are English majors— they perceive the 
English script more helpful, based on actual usage statistics, compared with that of the 
Chinese script. Gist, background information, and glossary (the smallest text unit) are 
used infrequently because they give only static textual information. The Gist, though 
with information for larger units than the other two, loses preference ^rom users 
because the text style is very formal and thus difficult to understand. For background 
information, another factor to consider is whether the users can see the relevance of it 
to the video. This can be verified because we have collected perception data by a 
questionnaire instrument in Table 4. The data show that only 35% of the twenty 
subjects reported the background information help was useful; apparently it was not 
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deemed relevant nor helpful. Although REPEAT and REPEAT_P_SENT give audio and 
visual information, the input unit is much smaller compared with that of BACKWARD. 
The results are summarized in Table 5. The actual use statistics showed that they were 
used infrequently. 
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Tables Comparison of Various Kinds of On-Line Help 

linguistic or processing factors 



1 level of function 


frequency 
of actual 
use 


text/audio/ 
visual 


unit size of 

process-ed 

input 


freq- 

reporte 

d%® 


ist Pause 


low 


visual/*more 
processing time 


0 


85% 


iSt oacKW 


highest 


audio/visual 


large 


85% 


2nd Chinese 


3rd 

highest 


Lltext/audio/ 
visual 


can be large 


55% 


2nd English 


2nd 
highest 


L2text/audio/ 
visual 


can be large 


80% 


2nd Gist 


low 


text 


large 


50% 


2nd Backg Info 


low 


text 


irrelevant 


35% 


2nd Glossary 


low 


text 


smallest 


80% 


2nd Repeat 


low 


audio/visual/ 
reiniorceineni 


small 


70% 


2nd Rep-P-sent 


low 


audio/visual/ 
^context 


small 


80% 



® the percentage here indicates the proportion of 20 subjects who reported useful for 
a particular kind of help in the questionnaire 

The high usefulness frequency reported in the Backward function, 85%, is 
reasonable and can be validated in subjects' actual use of the facility. But the one in 
Pause is strange because it was not used often, nor were the glossary. Repeat, and 
repeat the previous sentence functions. The high frequency reported in English script 
request could also be validated in it actual use, but the low frequency in Chinese script 
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request again was again puzzling. There is discrepancy between how often subjects 
actually used the help and how useful they felt. 

Functionally speaking, the first-level fodlities are more convenient to use than the 
second-level ones, and the two functions received the highest reported usefulness rates. 
If we can equate what learners actually use with what they believe useful, then two 
tentative conclusions can be drawn in this study. First, input with more modes is 
preferred than that with less, and thus used more often. Second, re-access of input 
with larger units is preferred to that of smaller, and thus accessed more often. 

In the second study (for detail, see Liou 1994a), 14 subjects with different 
English proficiency levels were reading an article in the same way as that in the paper 
version of Mosaic 11 except that all of the words displayed on the computer screen are 
linked to the entries of the Longman English-Chinese Dictionary of Contemporarv 
English (Longman, 1978). All the words consulted are recorded by the computer 
which help infer reading strategies (see Table 6 for records of two subjects). One 
important strategy found in Liou (1994a) is the strategy of re-reading local spots. To 
unitize the strategy data is like what other researchers have done: operationalize, or 
infer the frequency of strategy use based what we know about what learners are doing 
in CALL context. 

It is found that learners did use the glossary help to assist reading 
comprehenaon, but might not be able to use the optimal look-up strategies. She 
investigated three groups of EEL learners who read an article with the Longman 
bilingual English/Chinese dictionary entries linked to each of the words in the reading 
text. Examining the individual log files of words consulted by the computer, we could 
find the order of words looked up. Compatible to Hulstijn*s data (1993), most of them 
were linear within paragraphs or even within the passage. There were some incidents 
that subjects went back to look up words in the previous sentences to reconfirm their 
initial hypothesis, which was then defined as the strategy of re-reading local spots. As 
readers' proficiency decreased, the frequency of use of the re-reading local spots 
strategy increased. Namely, the strategy of re-reading local text strategy was found to 
differentiate the three groups of learners of various proficiency levels, but those of re- 
reading globally andjpre-reading did not tell various learner groups. Interestingly, two 
ineffective word consultation strategies (of the on-line Longman dictionary) were 
documented: ignoring key words, and choosing incorrect word senses in the multiple 
dictionary entries. The best learners were found to skip some key words and not look 
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them up in the dictionary. 

Table 6 Consulted Words in the Computer Log File (based on Mosaic 11: Reading) 



S3 

PR&READING 

irony 



DURING-READING 

archaeology 

gaps 

divine 

flung 

fling 

strands 

skeleton 

calcium 

compound 

titanium 

spares 

needle 

cutoff 



POST-READING 

downfall 



S5 

chill 
spine 
brevity 
^brevity 

archaeology 

gaps 

#gaps 

flame 

#flame 

divine 

#divine 

flung 

fling 

#flung 

#fling 

skeleton 

calcium 

titanium 

learnedly 

thermostat 

circuits 

kit 

#kit 

spares 

needle 

cutoff 

moral 
downfall 



Total words 16 
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#: word looked up the second time by the same subject 

It is interesting to note the process data shown in Table 6* S5 clicked on 
••flung,** the dictionary entry gives ••the past tense of fling^^ so she clicked on ••fling/ 
It seems that she could not flgure out the context and select a proper word sense, nor 
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retain it. So she went through the process again. Data of this type could be difficult 
to capture if without the use of the hypertext and the tracking programs. 

Limitations and the Prospective Solutions/Researcli Directions 

The major disadvantage of on-line hard data recorded by computing tools is that 
the learning materials are very much context-bound. The computer can record only 
particular lands of data, constrained by the types of interaction designed in the 
courseware. Further, it is crucial to find a universal framework to describe learner- 
computer interaction such as the discourse framework proposed by Chapelle (1990; 
1994), so the CALL profession can communicate and disseminate information. The state 
of the art for the taxonomies of C^LL activities vary greatly from one to another. The 
third limitation is that traditionally, it does not, or cannot record student own 
spontaneous explanation of their process. But the modem technology has come to 
solutions recently which can record users' voice input as well as visual images. 

Other Potential Research Agendas 

Nowadays, the technology has new potentials in multimedia or hypermedia-type 
courseware where students have much freedom navigating in the environment. Student 
decision-making, which demonstrates higher-level cognitive skills, and other highly 
individualistic use of hypermedia systems become issues which can again be observed. 
Depending on how sophisticated the hardware and software support an institute has, a 
researcher can collect a huge range of on-line hard data as assessment of student 
strategy use. 

Gay and Mazur (1993) summarize new multimedia tracking programs in five 
formats: customized tracking, records of written products, interaction histories, on-line 
recorder, and player piano. One possibility for such a range of on-line data is that users 
can view their previous steps in the program, plan future moves, and reflect upon work 
in progress if tracking data are made visible, namely, investigating learners' 
metacognitive moves. Meanwhile, since network use computing facilities become 
widespread, specific communication activities in the network system can be used as on- 
line hard data. For example, orientation, directives, clarification, social exchanges, 
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acknowledgements, and requests for information (Gay & Mazur, 1993) are important 
communicative acts in language learning settings (e. g.» Liou 199S studied such 
cooperative learning interaction). In an ambitious large-scale research project, Huang 
(1994) and his colleagues are trying to compute the degrees of concentration and 
willingness of learning in learners. To illustrate, the degree of concentration is obtained 
by dividing the effective session time (common session time minus ineffective session 
time, defined and examined by the talking protocols which contain few key words after 
the key-word matching procedure is examined) by the complete session time. He would 
use some fuzzy reasoning to figure out the parameters, together with others, to enable 
the network system with an intelligent computer manager which can provide adaptive 
teaching strategies. 

In the near future, multimedia tracking programs would allow users to review 
and annotate real-time records of their use, a variation on the think-aloud protocol 
technique. Second, new tracking systems evolving into multimedia formats have their 
accompanying expressive potential, amenable to transformation into other useful 
computer formats which aid both qualitative and quantitative analysis, such as 
spreadsheets. 

The tracking function can thus be expanded to fulfill applied linguists' research 
agendas. For example, in Liou (1994a) reading program which has the only simple 
capacity of recording glossary help, it can be complemented now by another facility in 
Liou's system (1995), reading time for each page in a hypertext program in addition 
to glossary use log. In Liou (1995), the IVD courseware is implemented to include 
training of strategies, and it can record the number of times when subjects access video 
material once or twice and time for each session, evidence of use of the advance 
organizer strategy. 

III. CONCLUSION 

The findings and data collection techniques in the paper provide researchers in 
other fields a new horizon where different sources of L2 learner data can be 
triangulated to help uncover the SLA phenomena. In the computer-assisted instruction 
(CAI) field, the techniques have been very commonly used. It is commonly believed 
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by CAI scholars that research using techniques that allow in-d^th analysis of how 
students are constructing knowledge as they work proves fruitful, indicating a new 
research approach. In a survey article, Gay and Mazur (1993) discuss several kinds of 
data that computer tracking systems may record: keystrokes, content items seen by the 
user, navigation strategies, and paths constructed. They point out benefits of such data 
for learners and courseware developers. Learners may use the data as input of their 
metacognitive strategies: view their previous steps, plan future move, or reflect upon 
work in progress. These functions facilitate CALL environments to train learners in 
using metacognitive strategies such as planning, evaluating, or monitoring. The 
developers can also use the data to determine the effectiveness, usability, and 
comprehensibility of the system. This helps develop a functional understanding of how 
the technology is used. In their paper. Gay and Mazur call for triangulation of both 
the tracking data and the perception data from learner users. For language learning, 
the use of several methods, each with their strengths and limitations, allow for cross- 
validation and thus result in more accurate assessment of strategies and learning 
process. It is argued in this paper that computer-based data collection methods are 
meant to complement other traditional methods, especially when the former are 
constrained. 

Computer tracking facilities have limitations: its value for research will be 
shaped by the breadth and depth of the research questions, namely, constrained only 
by human beings' intellectual inquiry. As well supported by Chapelle (1994) "technical 
capabilities for data collection in CALL environments far exceed our current theoretical 
and analytical capabilities for their description and interpretation" (p. 42). Why are we 
waiting and keeping from using such a powerful tool in our profession? 
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